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ABSTRACT 

At the University of Canberra, Australia, the design and architecture faculty are 
trialling a range of approaches to incorporating learning technologies in the first year 
foundation studio to improve student learning outcomes. For this study researchers 
collected information on students ’ access to their assignment information and feedback 
from the learning management system (LMS) to discover how the students engaged in 
the design process. 

The studio curriculum was designed to encourage students to engage in a convergence, 
divergence dynamic (Brown 2009, Thomas, Billsberry et al. 2014) in developing their 
own understanding of the design process. The staff tailored around points of 
convergence, online instruction, assessment tools and feedback in studio. We argue that 
using learning technologies in this way can improve intentionality at the beginning of 
semester, enhance students understanding of feedback and facilitate a more iterative 
approach to problem based learning in studio practice. 


INTRODUCTION 

Design and architecture education traditionally relies on personal interactions between tutor 
and student in a physical space called the studio. Increasingly in Australian universities, 
studio tutors are expected to adopt LMS for delivery of information and provision of feedback 
(MacKenzie and Hocking 2014). This approach to blended learning in studio is therefore 
worthy of investigation. While there is no consensus on the use of the term, blended learning 
(Funda 2011, Pekta§ and Giirel 2014), it is generally defined as the combination of traditional 
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delivery methods ( face to face) with online learning technologies to enhance teaching 
methods (Hyo-Jeong and Bonk 2010). This paper extends this definition to consciously 
incorporate a convergence, divergence dynamic (Brown 2009, Thomas, Billsberry et al. 2014) 
widely used in education, but also synonymous with design thinking to blend traditional ways 
of teaching design process into the curriculum. The blended learning component of the study 
is the incorporation of LMS and hand held devices to engage students is design thinking 
without explicitly teaching theories underpinning the design process they are undertaking. 
Technology in design education can provide students with rich and meaningful multimedia 
content that is contextually relevant and can be accessed and enacted upon (Bower, Howe et 
al. 2014). Technology can provide cognitive support for difficult tasks with pre-packaged 
learning experiences, allowing the user to control the speed, frequency and iteration of their 
access to content to suit their learning abilities. 

In this context, there is an ongoing need to investigate the pedagogical practices that are most 
suited to a design curriculum influenced by technologies. For example Van Haren (2010) 
argues technology should support the agency of students in enacting, developing and 
determining rather than passively accepting so that they can achieve a deeper understanding 
of subject matter. While this approach to learning is familiar in a studio environment, using 
technology for technologies sake should be critically evaluated in both learning and 
assessment. 

This project examines the question; how do LMS support students engaging in the design 
process? The question can be further broken down to; how do they use feedback to improve 
and to what extent do these learning technologies support the student to develop a design 
process around the convergence/divergence dynamic. Understanding how students engage 
with and benefit from different forms of feedback forms a key part of this enquiry. 


DESIGN THINKING AND FEEDBACK 

Design thinking is increasingly integrated throughout the curriculum of higher education 
institutions, particularly at post graduate level. Yet while there is consensus about the value of 
teaching design thinking, there is little consensus on how it should be taught (Wrigley and 
Straker 2015). In particular the problems are centred around the generalised approach to 
design thinking as a universal cognitive practice, ignoring how design is shaped by the role of 
institutions and socio-cultural developments over time (Kimbell 2011). As a result the value 
of design thinking in furthering creative output and generating innovation is poorly grounded 
in evidence-based practices (Dong 2015). Although design thinking lacks a formal definition, 
this paper adopts of a design thinking approach to curriculum delivery in first year design 
studio. In this case studio is not the place for learning design studies, but rather opportunity to 
develop design practice (Tonkinwise 2014). Rather than teaching the theory of how designers 
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think, the studio tutors incorporate learning technologies in order to scaffold the students’ 
projects and feedback in a way that encourages them to develop their comprehension and 
practice. 

Learning design practice presents students with their own anxieties. Tonkenwise (2014) 
argues this is unsurprising as studio demands that students harness design thinking to 
creatively traverse the domains of specialised knowledge yet design education tends to reflect 
what is done in professional practice; something students cannot comprehend in their early 
years of training. Underlying this anxiety, in addressing social problems such as 
sustainability, practical education (learning by doing) is counter intuitive to the more abstract, 
risk taking approaches that tend to generate more creative ideas, generating a larger solution 
space for progressing a design problem within its socio-technical context (Bleuze, Ciocci et 
al. 2014). 

Evaluation of design in the form of criticism also tends to value the students’ design work in 
terms of appearance or affect, privileging the high art content, history and theory courses in 
the curriculum of most schools and reflecting the research background of faculty staff. While 
some schools take a more sociological stance to design theory, engaging with the everyday 
cultural practices, such programs are in the minority (Tonkinwise 2014). In this way feedback 
in the form of criticism can reinforce the notions that students need to demonstrate adequate 
discipline knowledge more so than contextually relevant knowledge to the world we live in 
(Moore 2005). 

Research suggests feedback is the most powerful method of engaging with students, and can 
be used to improve learning (Hattie and Timperley 2007, Hattie 2009). But other research 
focused on the use of written feedback, seemingly contradicts this finding and states that 
students actually seldom access their feedback and learn very little from it for a number of 
reasons. These included a lack of understanding, relying on their memory of what was said, 
and because they are more focussed on their grades then on the feedback (Higgins 2000, 
Carless 2006, Weaver 2006). Recent research by Blair et.al. (2013) suggests that immediacy 
of feedback in written form, timely and accessible, and using a wider range of feedback 
mechanisms would enhance the student learning experience. 

There are a number of definitions of feedback, Hattie (2007) giving a very broad 
characterisation defined it as the “information provided by an agent about aspects of one’s 
performance or understanding” (p. 187). In a meta-analysis of over 134 studies on the use of 
feedback in education, he found that it was essential to the learning process and was among 
the most powerful influence on achievement (Hattie 2009). In an earlier work, Winne and 
Butler (1994) defined feedback from the perspective of the learner, as the “information with 
which a learner can confirm, add to, overwrite, tune, or restructure information in memory, 
whether that information is domain knowledge, meta-cognitive knowledge, beliefs about self 
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and tasks, or cognitive tactics and strategies” (p. 5740). Carless (2006) limited his use of the 
term feedback to the responses made on student assignments. For him, “it encompasses 
written annotations and comments on drafts or on finalised assignments, in addition to verbal 
dialogues prior to or after submission” (Carless 2006), although he goes on to argue that this 
form of feedback can serve multiple functions, for improving future assessments (Carless 
2007), to justify a grade, or even “the fulfilment of a ritual which is part of academic life” (p. 
220). In this paper we limit the meaning to the feedback, verbal or written, given to students 
for their assessment tasks. But also explore Winne and Butler’s understanding to see how 
much the learner in fact uses the feedback received to add to, fine tune, or change their 
responses. 

Hattie found that the value of feedback could be described as a powerful motivator and 
improver of learning, but Carless (2006) found in his study that most students were primarily 
motivated by marks and did not engage much with the written feedback they received (see 
also Crips, 2007). Weaver (2006) concurs, arguing that the value of feedback depended on 
the student’s individual notions and understanding of the written information, which may not 
be the same as their tutor or lecturer, in which case students would have a great deal of 
difficulty in using the feedback received to improve their learning. Higgins (2000) also found 
that many students were simply not able to understand written feedback or knew what to do 
with it, failing to understand the comments or alternatively misinterpreting them. This may 
particularly be true if the feedback comments are written as suggestions for improvement, 
which some students may take as literal, but others may take these as optional (Crisp 2007). 
In fact Crips argued that students seldom responded to feedback by making changes to their 
subsequent submissions, as suggested in the given feedback. 

This was a problem identified in our research, so in this study we explored whether students 
used feedback, and what if any impact this had on their subsequent results. Some researchers 
have suggested that the difference is in the quality, accessibility, timeliness, legibility and 
relational aspects of the feedback given (Chang, Watson et al. 2013). Similarly, in their study 
of e-feedback, McCabe, Doerflinger, and Fox (2011) found that students and staff perceptions 
were that “e-feedback procedures increase clarity of feedback compared to handwriting, save 
paper and ink resources, and result in faster and also better, more detailed feedback” (p. 178). 
However, Blair et al (2013) found that although students wanted quality feedback, that 
directed them in their learning, and that was given in a timely manner, they also found that 
whilst some students preferred written feedback, others preferred verbal feedback. This was 
because students found the verbal feedback was easier to understand, they were able to 
request further clarification, and because written feedback was often poorly constructed or 
used overly academic language. An assumption could be made here that students view 
feedback for no other purpose than to see what they need to improve or at least meet the 
learning outcomes of the activity, depending on whether they were deep or surface learners 
(Calvo and Ellis 2010). 
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In this study we asked students what type of feedback they preferred and sought to understand 
what it was about that feedback that they found improved their learning. The following 
section describes the methodology followed by the findings of these questions. 


METHODS 

This research used both qualitative and quantitative data to evaluate the effectiveness of LMS 
for providing feedback in a first year design studio. Student patterns of access to Moodle, the 
University LMS, were collected. Data included students’ frequency of access, timing of 
access compared to the release of project briefs, feedback (formative) and results (summative) 
for each project. Data was also collected data on student views of videos as recorded in 
EchoSystem, the lecture capture and video streaming system. Other data included student 
results collated from the rubric, and final grades following the final assessment of each 
project. Reflective summaries for each assessment were collected from the LMS following 
the final submission. Semi structured interviews were undertaken with studio tutors to 
evaluate their perception of the effectiveness and ease of use of the LMS for providing 
feedback. 


CASE STUDY 

The case study was an introductory design studio for students studying majors in architecture, 
landscape architecture, interior architecture and industrial design. The unit was delivered with 
a combination of online and face to face lectures. All studio work was undertaken in a 
conventional face to face format in a purpose built studio space. Students were delivered 
information and feedback via Moodle, the University LMS. Two tutors conducted the studios 
with approximately twenty students per class. Both tutors participated in the research and 
research ethics was granted for both surveying and interviewing students and staff. 

The curriculum developed for this studio is based on the design thinking approach of 
divergence and convergence (Brown 2009). The studio had three assessment stages (A01, 
A02 and A03) based on three interrelated and interdependent assignments that were 
scaffolded to create a final design piece incorporating all the assessment into a single final 
presentation. The three assignment themes; idea, form and object guided the students through 
a design thinking process based on the convergence/divergence dynamic (Ligure 1). 
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Figure 1 Design thinking process used in the research (adapted from Brown (2009)). 

Furthermore, the information for each assignment (project brief, project value, submission 
requirements, and assessment criteria) reflected the phases of a design process. The delivery 
of that information along with the provision of online feedback was timed to coincide with 
points of convergence in the students design thinking process (Figure 2). 



Figure 2 Design thinking process applied to the curriculum. 


To set the expectations in the studio, the staff used videos to explain the nature of studio and 
ways of learning. The importance of feedback has been reinforced with the former students’ 
experiences videos and the process was explained during lectures and studio classes. Students 
were informed of the nature of feedback and its importance before every formative feedback 
session at the beginning of the class. The design language was presented and explained in the 
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lectures and further enhanced during the verbal and written feedback. In this studio, tutors 
encouraged students to reflect on their feedback by extending the process using the formative 
and summative feedback and including various techniques of feedback, both face-to-face and 
online via the LMS. 


PROVISION OF FEEDBACK 

The feedback was provided in two stages: formative (feed forward) and summative (grade and 
short comment) (Table 1). During the studios where formative feedback was provided, 
students presented their work in front of their peers and received verbal feedback and a rubric 
with comments from their tutors using a touch screen on a hand held device (Figure 6). 
Students were explained their grade by the tutor using the rubric to help them understand 
discursively why they achieved the grade and how they can improve. The qualitative terms in 
the rubric text reflected the University assessment policy relating decryptions to grades. (E.g. 
satisfactory equates to a pass, excellent equates to a distinction) The verbal comments during 
the critique sessions were also recorded in form of the notes by a student peer. The students 
were provided an additional week to improve their assignment based on this feedback. The 
students were required to reflect on their formative feedback in 200 words and submit that in 
conjunction with their assignments. The refined work was then submitted using LMS, on 
which they would receive the summative feedback. 


Video views 


% 




Students 

Average total 

Video Title 

Duration (Minutes) 

viewing 

time 

Introduction to Design Studio 

3 

54% 

2.50 

Students on Design 

5 

35% 

2.49 

Light Tone and Colour 

20 

6% 

17.17 

Ordering Elements 

21 

21% 

15.68 

Point Line Plane 

26 

33% 

18.60 

Volume 

12 

27% 

11.10 


Table 1: Percentage of students viewing the videos and their average total time 


The most complete feedback process was in the second assignment, and thus we have selected 
those results for the purpose of this paper. In order to evaluate how effective the rubric 
feedback was in the process, the tutors were asked to provide verbal and written feedback 
differently. Tutor 1 adopted a personally tailored and more precise approach to written and 
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verbal feedback. Tutor 2 used general formulation using generic terms to describe the students 
work. 


DATA ANALYSIS 

Use data was collated for each student, descriptive statistics derived, and plotted in excel. The 
time series data was based on the date and time stamp information for all accesses of 
information by the students. 

The student reflections were collated, information de-identified, and a discourse analysis 
conducted looking at both the structure and practice of the language used by the students 
(Jorgensen and Phillips 2002). This enabled the research team to make observations about the 
language use in the context of the design discipline, and within the practice of the students. 
This allowed the researchers to tentatively make decisions about the students’ change in 
learning of design literacy, and help contribute to a general understanding of the process. 

The interview data from the tutors was analysed to identify how the students’ experience of 
receiving the feedback correlated with the tutors’ experience of providing the feedback. This 
triangulation helped us to understand the value of the immediacy of the feedback rather than 
focussing on the students’ comprehension. 


FINDINGS 

The findings from the research interrogate the students’ access to online information. The data 
is presented in two parts. The first part, resources, includes instructional videos and online 
lectures, and assessment instructions including assignment briefs and sample assessment 
rubrics. The second part, feedback, includes formative feedback in the form of written rubrics 
and comments, and summative, or grade only. 


STUDENT INSTRUCTIONS 

Throughout the course a number of resources were developed to support the students in their 
progress, these included short instructional videos, similar to the example videos described by 
Kay (2012), but with a focus on specific design elements and the use of these in the design 
process. Extensive documentation and instruction information was developed using the 
webpage tool in Moodle. Information included submission criteria, and how to submit the 
work electronically, as well as a sample of the assessment criteria. 


54 



A. MacKenzie, M. Muminovic, K. Oerlemans 


JPBLHE: VOL. 5, NO. 1, 2017 


VIDEOS 

In addition to the online lectures, two additional videos were included. The introduction video 
established the expectations for the assignments. The students on design video includes past 
students talking about their experience of the unit. These two videos better meet the definition 
of extra materials (Kay, 2014), but proved the most popular with the students. Students 
accessed these items much as expected, as is common across other disciplines (Danielson, 
Preast et al. 2014). Table la shows the percentage of students viewing the videos. The data 
indicates the instructional videos (1 and 2) provided on Moodle in the first week were the 
most popular. The aim of these videos was to help student prepare for studio practice, with 
practical tips and advice from previous students about how to work successfully. Videos 5 
and 6 included conceptual examples of successful project outcomes to help students 
understand what was expected. 

The videos are listed in order that they appear on the LMS unit site. As always, not all 
students will watch all videos. There were 137 students enrolled in the unit who viewed none, 
one or some of the videos, but not one video was viewed by all students. What is surprising 
and perhaps contrary to previous findings (Wiese and Newton 2013, Danielson, Preast et al. 
2014) is the length of views. Table lb shows that students who viewed the videos watched 
approximately 50% of videos 1 and 2 and between 70%- 92% of the remaining instructional 
videos. This could be an indication of the type of video, which fit more into the category of 
worked examples (Kay 2012), rather than the more common lecture capture videos. 




AOl 

A02 

A03 

Tutor 2 

formative 

86% 

71% 

79% 


summative 

93% 

90% 

81% 

Tutor 1 

formative 

100% 

95% 

75% 


summative 

100% 

95% 

90% 


Table 2: Submission o 


assessments Time 1 and Time 2 for each assessment task 


ASSESSMENT INSTRUCTIONS 

The team also examined the viewing patterns of the assignment instructions. Figure 4 shows 
the number of times the assignment instructions were viewed by students. Between 22 and 30 
students viewed the assignment instructions once on Moodle for each project. The viewing 
patterns for each project were similar. As expected students viewed assignment 1 instructions 
the most, but as they moved through the tasks, they showed less dependency on the 
instructions. 
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No. of times Assessment Instructions 
viewed by students 



Number of times viewed 


AOl 

- A02 

A03 


Figure 4 Number of times Assessment instructions viewed by students. 

More interesting for our study, is the findings in Figure 5, which shows a timeline of access of 
the views of the assignment instructions. The x axis shows the weeks of studio, including the 
assessment window. For example AOl formative feedback was given in the week starting 
7/3/16 and summative assessment was provided the following week beginning 14/3/16. The Y 
axis shows the frequency of students’ access to assessment instructions. For example AOl was 
viewed 56 times immediately after the first week of formative feedback. There is an ebb and 
flow showing how students access the assessment instructions in order to understand the 
project requirements. There are students who will view all the information, but there are also 
those students who only view what they need to as the tasks get closer to their due date. 
However, what the time series information shows us is that students engage in convergence 
moments prior to the submission of their assignments, in order to meet the requirements of the 
submission. We also see that students engage in convergence moments for subsequent 
submissions by viewing the instructions from the previous submission. 
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Timeline of access 


60 

50 

40 

30 

20 

10 

0 

8 / 2/16 15 / 2/16 22 / 2/16 29 / 2/16 7 / 3/16 14 / 3/16 21 / 3/16 28 / 3/16 4 / 4/16 11 / 4/16 18 / 4/16 25 / 4/16 2 / 5/16 9 / 5/16 16 / 5/16 23 / 5/16 30 / 5/16 

• A02 views • AOl views ▲ A03 views 



Figure 5 Timeline of access of Assessment Instructions. 


STUDENT ACCESS PATTERNS FOR FEEDBACK 

The previous section provides an overall picture of how students access information during 
the semester. This section looks more closely at how students access feedback. Formative 
feedback (or feed forward) was provided three times during the semester in studio using an 
online rubric on a handheld touch screen smart device (Figure 6). The students were provided 
both verbal and rubric feedback in class and given an additional week to improve their 
assignment prior to receiving a summative grade. 
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Figure 6- Sample rubric from hand held device screen 


Looking more closely at assignment 2, students viewed the rubric during the week between 
the formative and summative feedback did so in order to get as much feedback as possible for 
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improving their assignment. There were 91 views of the rubric prior to the submission of 
Assessment 2 after the in studio formative feedback. There were an additional 289 views of 
the rubric in total during the week between the formative and summative feedback. We can 
assume that students who viewed the rubric once were only looking for their results; however, 
there can be an assumption that those who viewed it more than once were reviewing the result 
and seeking further feedback from the rubric in order to improve their outcomes. Table 3 
shows the relative improvement in grades. Tutor 1 gave rubric, comments and verbal 
feedback whereas tutor 2 relied on rubric only for formative feedback (Table 3). There is no 
evidence that the rubric by itself caused an improvement in results, as can be seen from the 
table below. Despite the demonstrated improvements from tutor 1 student outcomes, 
compared to tutor 2, is difficult to determine what caused the improvement, as students in 
both groups frequently accessed the rubric feedback in the week following the formative 
feedback. It may well be the combination of the 3 forms of feedback that is the most 
powerful, with the verbal on the day and the written and rubric feedback given online for 
students to refer back to regularly as they seek to improve their outcomes. 



A02 Formative 

A02 Summative 

Tutor 1 - verbal, written and rubric 

57.62% 

66.86% 

Tutor 2 — rubric 

62.74% 

62.82% 


Table 3: Average marks for Time 1 and Time 2 for Assessment 2 for each tutor 


DISCUSSION 

The aim of this research was to provide evidence of students’ engagement with online 
information and feedback in a design studio. In addition, the team were interested in how the 
timing of feedback could coincide with the students’ cognitive process of creative exploration 
(divergence) and design resolution and communication (convergence). The design of the 
curriculum followed a divergent/ convergent process broadly determined by the timing of 
assignments submissions and timeliness of feedback. The team consciously designed the 
curriculum to focus students’ attention around points of convergence as they progressively 
developed their studio assignments. 

The focus on points of convergence for data collection served two purposes. Firstly the 
students’ patterns of access in terms of frequency and over time could be mapped. Secondly 
the students were required to progressively build on each assignment and in doing so 
creatively engage in a design process that involved both iteration and engaging (though not 
consciously) in the convergence divergence dynamic. It was hoped that by targeting feedback 
leading up to convergence points in the assessment rather than providing weekly feedback, 
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students would be more likely to undertake more divergent thinking in the weeks when 
feedback wasn’t given. 

While the evidence of convergence was clear, the evidence of divergence was more difficult 
to determine, but we suggest the student’s execution of the assignments and scaffolding of 
each submission based on the previous assignment shows improvement in their ability to 
complete the assignments. Tutors also reported that the students were also prepared to make 
significant conceptual changes to their final submissions in the week between the formative 
and summative feedback in order to achieve improvement. This suggests students could 
comprehend the feedback and revisit their project in a manner that mimics a more iterative 
design process. As mentioned in the results we can deduce that the frequency of views of 
assessment instructions coincided with period of work for students that preceded an 
assessment event. Notably the periods of exploration following the launch of the projects 
coincide with relatively low levels of access to the LMS in order to read assessment 
instructions. We would argue that while the data may suggest the students were not engaged 
in the assessments during this period due to the low level of LMS access, they were engaged 
in a form of divergence and information seeking in other ways as part of studio practice. We 
would argue that the early phase of the design cycles allowed the students to learn divergently 
by exploring without the constraints imposed by assessment criteria. For the team this 
tentative finding would suggest future research should focus on when not to give students 
feedback or assessment information in order to encourage the risk taking and creative leaps 
consistent with divergent thinking. 

Tonkenwise (2014) argues that students find the creative leaps required in design practice to 
be daunting. Therefore encouraging creative exploration through some form of strategic 
retreat from the student tutor interaction may prove to be useful in complimenting this 
research to achieve better student outcomes. 

In the first phase of this research in 2015, we reported that students preferred verbal feedback; 
however students who received and accessed other forms of feedback demonstrated the 
greatest rate of improvement in their grades. Similarly, students who revisited the project 
information, including accessing previous assignments information for subsequent 
submissions also demonstrated the greatest improvements. This research is instructive for 
studio curriculum designers and tutors who want to maximise the efficacy of online 
information and student interaction with LMS. The concept of traditional weekly studio 
verbal feedback is both labour intensive and inefficient in terms of students and academic 
staff time. It could also be argued that the efficacy of all forms of feedback is most evident 
during period of convergence. Similarly, the points of convergence in the semester are 
relatively short and focussed leaving large period of time to allow students to explore 
different ideas and approaches. However, while the use of LMS allows students to access 
their feedback immediately and revisit the information outside of studio time, it remains to be 
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seen how this method can encourage students to progress their design process without 
accessing LMS when divergence is needed. 

Tutors should take comfort in changing their patterns of feedback to allow students more time 
to indulge in divergent processes in between assignments. Similarly students have more 
flexibility in how they access assignment information and feedback. As Universities demand 
more flexible and intensive modes of delivery, studio tutors can make the most of face to face 
time leading up to assessment periods and rely more on LMS to support students learning at 
other times. 


CONCLUSION 

By 2013, nearly 18 percent of students in Australia studied off campus, with a further 9 
percent choosing to complete at least some of their study online (Norton and Cherastidtham 
2014). The implications of this growing trend and demand for use of LMS to support, 
enhance or replace the more traditional modes of face-to-face teaching at universities 
(Laurillard 2013) including the adoption of new pedagogies, increased demands on academics 
time, and changing student expectations, and are well established (Bonk 2009). It is 
unsurprising that university programs in design education are under pressure to expand the 
adoption of LMS into the design studio. 

While this project doesn’t explicitly teach students about creative thinking or design process, 
it engages students in a design process through a curriculum based on the convergent 
divergent dynamic. The value to design educators is that a curriculum designed around a 
creative thinking approach, such as the one we have used, does help students to better 
understand and adopt a creative process in learning about design. By consciously 
incorporating LMS into the learning process for students, they can achieve positive outcomes 
that enhance more conventional forms of face to face verbal feedback. Studio tutors can use 
LMS to gain a better understanding of how students engage in design outside of formal studio 
interactions and better target the use of feedback. The research suggests that design tutors 
should focus on the diversity and timeliness of project information and feedback based on the 
convergent divergent dynamic in order to achieve better result from students at the beginning 
of their studio learning journey. 

This research focussed on the patterns of access to online data, it revealed levels of 
engagement in information but not their levels of comprehension of the assignment 
requirements. Further research to better understand how students comprehended both the 
words used in the instructions and feedback rubrics may help to improve our understanding of 
how student progress between convergence points. 
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